SELF-ATTENTION MECHANISM

Enhancing Self-Attention with Knowledge-Assisted Attention Maps (2022.naacl-main)

Challenge:	Recent years have witnessed substantial progress in the development of neural ranking networks, but an increasingly heavy computational burden due to growing numbers of parameters and the adoption of model ensembles.
Approach:	They propose a two-stage distillation method that allows a smaller student model to be trained while benefiting from the better performance of the teacher model.
Outcome:	The proposed method shows higher-quality rankings compared to the teacher model.

Copied to clipboard

Jiangang Bai, Yujing Wang, Hong Sun, Ruonan Wu, Tianmeng Yang, Pengfei Tang, Defu Cao, Mingliang Zhang1, Yunhai Tong, Yaming Yang, Jing Bai, Ruofei Zhang, Hao Sun, Wei Shen

MM-GATBT: Enriching Multimodal Representation Using Graph Attention Network (2022.naacl-srw)

Challenge:	Existing works of knowledge infusion depend on multi-task learning frameworks, which are inefficient and require large-scale retraining when new knowledge is considered.
Approach:	They propose a method which integrates knowledge-generated attention maps into the self-attention mechanism and integrates it into the model.
Outcome:	The proposed model outperforms existing methods on academic datasets and industry-scale ad relevance applications.

Copied to clipboard

Seung Byum Seo, Hyoungwook Nam, Payam Delgosha

Our Neural Machine Translation Systems for WAT 2019 (D19-52)

Challenge:	Existing models that use a self-attention mechanism to create graphs with multiple modes ignore interaction between entities, multimodalities, or both.
Approach:	They propose a multimodal graph representation learning model that captures relational semantics within one modality and interactions between different modalities.
Outcome:	The proposed model outperforms existing models on the MM-IMDb dataset in all aspects of multimodal representation.

Copied to clipboard

Wei Yang, Jun Ogata

Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024.starsem-1)

Challenge:	In the last five years, statistical machine translation is gradually fading out in favor of neural machine translation.
Approach:	They describe a novel Neural Machine Translation (NMT) system for the WAT 2019 translation tasks they focus on.
Outcome:	The proposed system improves translation accuracy while replacing absolute position representations with relative positions.

Copied to clipboard

Dongjun Jang, Jinwoong Kim, Hyopil Shin

Sentiment Analysis

Challenge:	Existing methods to inject lexical features into self-attention mechanisms have shown remarkable performance across various downstream tasks in NLP.
Approach:	They propose to inject lexical features into the self-attention mechanism of Transformer-based models by injecting lexicon-based Sentiment Lexical Attention into the attention scores throughout the training process.
Outcome:	The proposed method shows significant performance improvements on the NSMC sentiment classification benchmark and is able to perform in out-of-domain tasks.

Self-Attention Graph Residual Convolutional Networks for Event Detection with dependency relations (2021.findings-emnlp)

Copied to clipboard

Anan Liu, Ning Xu, Haozhe Liu

Detecting Ambiguous Utterances in an Intelligent Assistant (2024.emnlp-industry)

Challenge:	Existing methods to classify events using syntactic dependency relations have not been developed.
Approach:	They propose a model which combines syntactic dependency relations with attention-based dynamic tensors to mine node-to-node latent dependency relations via self-attention mechanism.
Outcome:	The proposed model improves on the ACE2005 dataset and compares with baseline models.

Copied to clipboard

Satoshi Akasaki, Manabu Sassano

Challenge:	ambiguous utterances can be interpreted as either chat or task intents in intelligent assistants . ambiguity of intent is particularly noticeable in intelligent devices where task-oriented and non-task-oriented utterrances are mixed and most utterations are short due to characteristics of devices.
Approach:	They propose to feed sentence embeddings developed from microblogs and search logs with a self-attention mechanism to detect ambiguous utterances robustly.
Outcome:	The proposed model outperforms baselines and a strong LLM-based model.

Adaptive Attention Span in Transformers (P19-1)

Copied to clipboard

Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin

Obligation and Prohibition Extraction Using Hierarchical RNNs (P18-2)

Challenge:	We extend the maximum context size of a neural network called Transformer to 8k characters.
Approach:	They propose a self-attention mechanism that can learn its optimal attention span . this allows for models with longer context and the capability to catch longer dependencies.
Outcome:	The proposed model achieves state-of-the-art performance on text8 and enwiki8 using 8k characters with no loss of performance, and maintains control over memory footprint and computational time.

Copied to clipboard

Ilias Chalkidis, Ion Androutsopoulos, Achilleas Michos

Label-Specific Document Representation for Multi-Label Text Classification (D19-1)

Challenge:	Existing methods for contract element extraction and contract element classification focus on indicative tokens, but they are not as efficient as the current ones.
Approach:	They propose a self-attention mechanism that converts each sentence to an embedding and processes the embeddables to classify each sentence.
Outcome:	The proposed method outperforms the flat BILSTM classifier even when it considers surrounding sentences because it has a broader discourse view.

Copied to clipboard

Lin Xiao, Xin Huang, Boli Chen, Liping Jing

Information Retrieval

Challenge:	Existing methods to classify documents using labels only assign one label to document . multi-label text classification is a challenging task because of the huge amount of documents, words and labels.
Approach:	They propose a Label-Specific Attention Network (LSAN) to learn a label-specific document representation.
Outcome:	The proposed model outperforms state-of-the-art methods on four datasets . it can predict low-frequency labels, and it can be used in sentimental analysis .

Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity (2022.tacl-1)

Copied to clipboard

Yiding Hao, Dana Angluin, Robert Frank

The Role of Global and Local Context in Named Entity Recognition (2023.acl-short)

Challenge:	averaging hard attention is used to recognize formal languages that UHAT and GUHAT cannot recognize.
Approach:	They analyze three formal Transformer encoders that differ in the form of their self-attention mechanism . they find that UHAT and GUHAT Transformers can only recognize formal languages in AC0 .
Outcome:	The proposed models can recognize languages that UHAT and GUHAT cannot . the proposed models are based on the DYCK and PARITY languages .

Copied to clipboard

Arthur Amalvy, Vincent Labatut, Richard Dufour

Self-Attention with Relative Position Representations (N18-2)

Challenge:	Named Entity Recognition (NER) models are usually applied sequentially because of their complexity.
Approach:	They explore the impact of global document context on Named Entity Recognition . they find that correctly retrieving global document contextual has a greater impact .
Outcome:	The proposed model can retrieve global context better than leveraging local context . authors say the model can push the state of the art further .

Copied to clipboard

Peter Shaw, Jakob Uszkoreit, Ashish Vaswani

CNNBiF: CNN-based Bigram Features for Named Entity Recognition (2021.findings-emnlp)

Challenge:	Recent approaches to sequence to sequence learning leverage recurrence, convolution, attention or combination of recurrent and convolutional neural networks.
Approach:	They propose an approach that extends the self-attention mechanism to consider representations of relative positions, or distances between sequence elements.
Outcome:	The proposed approach yields 1.3 BLEU and 0.3 BLUE on translation tasks . it is based on a relation-aware self-attention mechanism that can generalize to arbitrary graph-labeled inputs.

Copied to clipboard

Chul Sung, Vaibhava Goel, Etienne Marcheret, Steven Rennie, David Nahamoo

Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders (2024.findings-eacl)

Challenge:	Named entity recognition tasks require a self-attention mechanism with unconstrained length that fails to capture local dependencies.
Approach:	They propose a joint training objective which better captures the semantics of words corresponding to the same entity by augmenting the objective with a group-consistency loss component.
Outcome:	The proposed model achieves a test F1 of 93.98 with a single transformer model.

Copied to clipboard

Yingji Zhang, Danilo Carvalho, Marco Valentino, Ian Pratt-Hartmann, Andre Freitas

Temporal Attention for Language Models (2022.findings-naacl)

Challenge:	Recent work has struggled to achieve consistent results due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism.
Approach:	They propose a model that leverages the controllability of VQVAE to guide the self-attention mechanism in Transformer-based VAEs to improve semantic control and generation.
Outcome:	The proposed model outperforms existing state-of-the-art VAE models in terms of control and preservation of semantic information across different tasks.

Copied to clipboard

Guy D. Rosin, Kira Radinsky

Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)

Challenge:	Pretrained language models are trained on corpora derived from the web, but ignore this information.
Approach:	They propose a time-aware self-attention mechanism that captures time-specific contextualized word representations and allows the transformer to capture this information.
Outcome:	The proposed model achieves state-of-the-art on three datasets in different languages (English, German, and Latin) that vary in time, size, and genre.

Copied to clipboard

Ivan Kobyzev, Abbas Ghaddar, Dingtao Hu, Boxing Chen

Dependency Position Encoding for Relation Extraction (2022.findings-naacl)

Challenge:	Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise.
Approach:	They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention.
Outcome:	The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks.

Copied to clipboard

Qiushi Guo, Xin Wang, Dehong Gao

Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection (2020.findings-emnlp)

Challenge:	Existing methods to extract relation extraction from sentence are limited in focusing on leveraging dependency information.
Approach:	They propose dependency position encoding (DPE) that incorporates dependency connections and dependency types into the self-attention mechanism to distinguish the importance of different word dependencies.
Outcome:	The proposed method significantly outperforms the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.

Copied to clipboard

Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, Weiping Wang

Sentiment Analysis

Challenge:	Existing methods for sarcasm detection ignore the incongruity character in sarcasm, which is often manifested between modalities or within modalités.
Approach:	They propose to capture inter-modality incongruity in a text-based model by using a self-attention mechanism and a co-attention model to model the contradiction within the text.
Outcome:	The proposed model achieves state-of-the-art on a public multi-modal sarcasm detection dataset.

In-Context Former: Lightning-fast Compressing Context for Large Language Model (2024.findings-emnlp)

Copied to clipboard

Xiangfeng Wang, Zaiyi Chen, Tong Xu, Zheyong Xie, Yongyi He, Enhong Chen

Multi-grained Named Entity Recognition (P19-1)

Challenge:	Existing methods to reduce inference costs of transformer-based large language models entail quadratic complexity . et al., 2017): transformer-derived large language model performance is a major challenge.
Approach:	They propose a method that compresses long contexts into short soft prompts . they use the self-attention mechanism of the large model to extract and condense information .
Outcome:	The proposed method reduces compression costs by 68 to 112 times while achieving 90% of baseline performance.

Copied to clipboard

Congying Xia, Chenwei Zhang, Tao Yang, Yaliang Li, Nan Du, Xian Wu, Wei Fan, Fenglong Ma, Philip Yu

Hero-Gang Neural Model For Named Entity Recognition (2022.naacl-main)

Challenge:	Existing approaches treat Named Entity Recognition (NER) as a sequence labeling task.
Approach:	They propose a framework for Multi-Grained Named Entity Recognition where multiple entities or entity mentions in a sentence could be non-overlapping or totally nested.
Outcome:	The proposed framework outperforms current state-of-the-art frameworks by 4.4% in terms of the F1 score among nested/non-overlapping NER tasks.

Copied to clipboard

Jinpeng Hu, Yaling Shen, Yang Liu, Xiang Wan, Tsung-Hui Chang

Syntactically Aware Cross-Domain Aspect and Opinion Terms Extraction (2020.coling-main)

Challenge:	Named entity recognition (NER) is a fundamental and important task in natural language processing.
Approach:	They propose a novel Hero-Gang Neural structure to leverage both global and local information to promote NER by using a Transformer-based encoder and a Gang module.
Outcome:	The proposed model can extract local features and position information from the Hero and Gang modules, and it performs on multiple datasets.

Copied to clipboard

Oren Pereg, Daniel Korat, Moshe Wasserblat

DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures (2026.eacl-long)

Challenge:	Supervised-learning approaches fail to scale across domains where labeled data is lacking.
Approach:	They propose a method for incorporating external linguistic knowledge into a self-attention mechanism coupled with a transformer-based model.
Outcome:	The proposed method enables leveraging syntactic knowledge from transformer-based models to bridge the gap between domains.

Copied to clipboard

Benno Uthayasooriyar, Antoine Ly, Franck Vermet, Caio Corro

Recurrent Attention Networks for Long-text Modeling (2023.findings-acl)

Challenge:	Existing models that take text block positions into account are not efficient for document understanding.
Approach:	They propose a layout-aware BERT model that takes into account text block positions in relative polar coordinate system rather than the Cartesian one.
Outcome:	The proposed model eliminates the need for absolute positional embeddings on a dataset more than six times smaller than the widely used IIT-CDIP corpus.

Copied to clipboard

Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, Qing Li

Multimodal Phased Transformer for Sentiment Analysis (2021.emnlp-main)

Challenge:	Existing approaches to encoding long documents using self-attention have been limited by quadratic computational complexities and limited application in long text processing.
Approach:	They propose a long-document encoding model that allows the recurrent operation of self-attention.
Outcome:	The proposed model extracts global semantics in token-level and document-level representations, making it inherently compatible with both sequential and sequential tasks.

Copied to clipboard

Junyan Cheng, Iordanis Fostiropoulos, Barry Boehm, Mohammad Soleymani

Token-level Dynamic Self-Attention Network for Multi-Passage Reading Comprehension (P19-1)

Challenge:	Existing methods to model multimodal sentiment analysis are limited due to their complexity and memory footprint.
Approach:	They propose a multimodal Sparse Phased Transformer to reduce self-attention complexity and memory footprint.
Outcome:	The proposed method achieves comparable or superior performance with a 90% reduction in the number of parameters.

Copied to clipboard

Yimeng Zhuang, Huadong Wang

Machine Reading Comprehension

Challenge:	Multi-passage reading comprehension requires the ability to combine cross-passages information and reason over multiple passages to infer the answer.
Approach:	They propose a Dynamic Self-attention Network (DynSAN) which processes cross-passage information at token-level and meanwhile avoids substantial computational costs.
Outcome:	The proposed model achieves state-of-the-art performance on the SearchQA, Quasar-T and WikiHop datasets and further ablation validates the effectiveness of its components.

Nonparametric Forest-Structured Neural Topic Modeling (2022.coling-1)

Copied to clipboard

Zhihong Zhang, Xuewen Zhang, Yanghui Rao

Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction (D18-1)

Challenge:	Existing hierarchical neural topic models can only extract topics at the same level.
Approach:	They propose to use self-attention mechanism to capture parent-child topic relationships and build a sparse directed acyclic graph to form a topic forest.
Outcome:	The proposed model outperforms baseline models on topic hierarchical rationality and affinity.

Copied to clipboard

Jinhua Du, Jingguang Han, Andy Way, Dadong Wan

Prompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptation (2023.acl-long)

Challenge:	Existing approaches to label large-scale data are inadequate for distantly supervised relation extraction (DS-RE).
Approach:	They propose a multi-level structured (2-D matrix) self-attention mechanism for DS-RE using bidirectional recurrent neural networks.
Outcome:	The proposed framework significantly outperforms baselines on two publicly available DS-RE datasets in terms of PR curves, P@N and F1 measures.

Copied to clipboard

Ibrahim Taha Aksu, Min-Yen Kan, Nancy Chen

A Survey of Retentive Network (2026.findings-acl)

Challenge:	Parameter-Efficient Transfer Learning (PETL) has the potential to address this problem, but it has yet to be applied to the zero-shot domain adaptation.
Approach:	They propose to use descriptions of target domain slots to generate dynamic prefixes that are concatenated to the key and values at each layer’s self-attention mechanism.
Outcome:	The proposed method outperforms previous methods on the MultiWOZ and SGD benchmarks.

Copied to clipboard

Haiqi Yang, Zhiyuan Li, Yi Chang, Yuan Wu

Challenge:	Existing studies on the effectiveness of the Retentive Networks have not yet been conducted.
Approach:	They propose a retention mechanism that integrates the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models.
Outcome:	The proposed retention mechanism combines the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models.

Original Semantics-Oriented Attention and Deep Fusion Network for Sentence Matching (D19-1)

Copied to clipboard

Mingtong Liu, Yujie Zhang, Jinan Xu, Yufeng Chen

Challenge:	Sentence matching is a key issue in natural language inference and paraphrase identification.
Approach:	They propose a semantics-oriented attention and deep fusion network (OSOA-DFN) that is oriented to the original semantic representation of another sentence and propagates attention information at each matching layer.
Outcome:	The proposed model can model sentence matching more precisely on three sentence matching benchmark datasets.

Modelling Context and Syntactical Features for Aspect-based Sentiment Analysis (2020.acl-main)

Copied to clipboard

Minh Hieu Phan, Philip O. Ogunbona

Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)

Challenge:	Existing approaches to aspect-based sentiment analysis do not fully leverage syntactical information.
Approach:	They propose an end-to-end aspect-based sentiment analysis solution that integrates syntactical information with part-of-speech embeddings and dependency-based embeddables to enhance the performance of the aspect extractor.
Outcome:	The proposed solution outperforms the state-of-the-art models on SemEval-2014 dataset in both subtasks.

Copied to clipboard

Zhiheng Huang, Davis Liang, Peng Xu, Bing Xiang

Improved Dependency Parsing using Implicit Word Connections Learned from Unlabeled Data (D18-1)

Challenge:	Existing methods for generating position embeddings are not fully utilized in NLP tasks.
Approach:	They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models .
Outcome:	The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models .

Copied to clipboard

Wenhui Wang, Baobao Chang, Mairgup Mansur

Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation (2021.naacl-main)

Challenge:	Pre-trained word embeddings and language models cannot capture word connections in a sentence.
Approach:	They propose to implicitly capture word connections from unlabeled data by word ordering model with self-attention mechanism.
Outcome:	The proposed model achieves 96.35% UAS and 95.25% LAS on the English PTB dataset.

Copied to clipboard

Kyeongpil Kang, Kyohoon Jin, Soyoung Yang, Soojin Jang, Jaegul Choo, Youngbin Kim

Machine Translation

Challenge:	voluminous historical records are difficult to fully utilize since they are written in ancient languages and some parts are damaged over time.
Approach:	They propose a multi-task learning approach to restore and translate historical documents using a self-attention mechanism.
Outcome:	The proposed approach improves the accuracy of the translation task over baselines without multi-task learning.

Evaluating Sentence Segmentation in Different Datasets of Neuropsychological Language Tests in Brazilian Portuguese (2020.lrec-1)

Copied to clipboard

Edresson Casanova, Marcos Treviso, Lilian Hübner, Sandra Aluísio

Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT (2022.acl-long)

Challenge:	Using automated analysis of connected speech is a promising direction for diagnosing cognitive impairments.
Approach:	They propose to use a novel model to segment impaired speech transcriptions . they propose to include a Linear Chain CRF and a self-attention mechanism .
Outcome:	The proposed system performs better than the existing model with three new datasets used to diagnose cognitive impairments.

Copied to clipboard

Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, Xiaodong He

Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models (2025.findings-emnlp)

Challenge:	Transformer-based pre-trained models achieve state-of-the-art results, but they can be prohibitively costly.
Approach:	They propose a fine- and coarse-granularity hybrid self-attention that shortens the computational sequence length in self- attention by progressively shortening the computational time.
Outcome:	The proposed model reduces computation cost by shortening the computational sequence length in self-attention.

Copied to clipboard

Zheyu Zhang, Shuo Yang, Bardh Prenkaj, Gjergji Kasneci

Cluster-Former: Clustering-based Sparse Transformer for Question Answering (2021.findings-acl)

Challenge:	Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs.
Approach:	They propose a method that explicitly integrates sparse dependency graphs into LLMs’ attention mechanism.
Outcome:	The proposed method outperforms existing LLM-based approaches by up to 12% on complex datasets while achieving competitive results with state-of-the-art approaches in synthetic data quality.

Copied to clipboard

Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, Jingjing Liu

Text Clustering

Challenge:	Existing models for encoding long sequences in deep learning suffer from high latency and memory demands.
Approach:	They propose a clustering-based sparse Transformer framework to perform attention across chunked sequences.
Outcome:	The proposed framework achieves state-of-the-art on several major QA benchmarks.

Fine-tune BERT with Sparse Self-Attention Mechanism (D19-1)

Copied to clipboard

Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang

ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation (P19-1)

Challenge:	Existing sparse self-attention fine-tuning models have been used to improve sentiment analysis, question answering, and natural language inference tasks.
Approach:	They propose a Sparse Self-Attention Fine-tuning model which integrates sparsity into self-attention mechanism to enhance the fine-tune performance of BERT.
Outcome:	The proposed model outperforms the baseline models on sentiment analysis, question answering, and natural language inference tasks and is able to interpret the input better.

Copied to clipboard

Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo, Xueqi Cheng

Dialogue Systems & Conversational Agents

Challenge:	Existing hierarchical recurrent encoder-decoder models treat all contexts indiscriminately, which may hurt the following response generation process.
Approach:	They propose a hierarchical recurrent encoder-decoder model that treats all contexts indiscriminately and uses a word level LSTM encoder to obtain the initial representation of each context.
Outcome:	The proposed model outperforms baseline models on Chinese customer services and English Ubuntu dialogue datasets in terms of both metric-based and human evaluations.

Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers (2025.findings-emnlp)

Copied to clipboard

Yingyu Liang, Heshan Liu, Zhenmei Shi, Zhao Song, Zhuoyan Xu, Jiale Zhao, Zhen Zhuang

Dynamic Structured Neural Topic Model with Self-Attention Mechanism (2023.findings-acl)

Challenge:	a large computational cost for attention computation in large language models is a major obstacle .
Approach:	They propose a convolution-like structure for attention computation using convolution matrices . they then propose an efficient approximation method to approximate the attention matrix .
Outcome:	The proposed method achieves nearly linear time complexity in n1+o(1) time.

Copied to clipboard

Nozomu Miyamoto, Masaru Isonuma, Sho Takase, Junichiro Mori, Ichiro Sakata

Attention-based Conditioning Methods for External Knowledge Integration (P19-1)

Challenge:	Recent topic models that capture the time-series evolution of topics assume that topics evolve independently without interaction.
Approach:	They propose a dynamic structured neural topic model which captures topic dependencies while capturing their dependencies.
Outcome:	The proposed model outperforms a prior dynamic embedded topic model regarding perplexity and coherence while maintaining sufficient diversity across topics.

Copied to clipboard

Katerina Margatina, Christos Baziotis, Alexandros Potamianos

Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text (P19-1)

Challenge:	Existing approaches for incorporating external knowledge into deep neural networks (RNNs) lexicon features are used to concatenate external information into the input or hidden network layers.
Approach:	They propose a method for conditioning external knowledge into RNNs by concatenating a representation of the external information to the input or hidden network layers.
Outcome:	The proposed approach improves performance on six benchmark datasets.

Copied to clipboard

Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, Marius Kloft

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model (2022.findings-emnlp)

Challenge:	Existing methods for unsupervised anomaly detection use pre-trained word embeddings . proper text representation is critical for designing well-performing machine learning algorithms .
Approach:	They propose a new anomaly detection method that builds upon word embedding models to learn multiple sentence representations that capture multiple semantic contexts via the self-attention mechanism.
Outcome:	The proposed method performs on Reuters, 20 Newsgroups, and IMDB Movie Reviews datasets.

Copied to clipboard

Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, Shinji Watanabe

Deep Reinforcement Learning-based Dialogue Policy with Graph Convolutional Q-network (2024.lrec-main)

Challenge:	Existing approaches to connectionist temporal classification (CTC) are based on pre-trained language models (LMs)
Approach:	They propose a formulation of connectionist temporal classification that relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through explicit output dependency.
Outcome:	The proposed model improves over conventional approaches across variations in speaking styles and languages while maintaining CTC’s training efficiency.

Copied to clipboard

Kai Xu, Zhengyu Wang, Yuxuan Long, Qiaona Zhao

Natural Language Interfaces

Challenge:	Existing methods for deep reinforcement learning lack the ability to learn the relationship between dialogue states and actions.
Approach:	They propose a graph-structured dialogue policy framework for task-oriented dialogue systems that uses bipartite graphs to construct two different bipartites and generate user-related and knowledge-related subgraphs.
Outcome:	The proposed framework significantly improves the effectiveness and stability of dialogue policies.

A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding (D18-1)

Copied to clipboard

Changliang Li, Liang Li, Ji Qi

Composition, Attention, or Both? (2022.findings-emnlp)

Challenge:	Spoken language understanding (SLU) involves intent determination and slot filling . existing joint learning methods only consider joint learning by sharing parameters on surface level rather than semantic level.
Approach:	They propose a self-attentive model to fully utilize the semantic correlation between slot and intent.
Outcome:	The proposed model outperforms existing methods in both intent detection and slot filling tasks on ATIS benchmarks.

Copied to clipboard

Ryo Yoshida, Yohei Oseki

Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization (2021.findings-acl)

Challenge:	Existing work suggests that language models implicitly learn syntactic structures of natural language, even though they do not receive explicit syntatic supervision.
Approach:	They propose a novel architecture that recursively compose subtrees with a composition function and selectively attend to previous structural information with sc-attention mechanisms.
Outcome:	The proposed architecture can induce human-like syntactic generalization by recursive composition and selective attention to previous structural information.

Copied to clipboard

Shuaiqi Liu, Jiannong Cao, Ruosong Yang, Zhiyuan Wen

A Transformer-based Approach for Source Code Summarization (2020.acl-main)

Challenge:	Existing models do not consider key phrases in determining attention weights of self-attention . Existing work does not consider the importance of key phrases when determining weights .
Approach:	They propose a model with highlighting mechanism to assign greater attention weights to key phrases . they propose two structures of highlighting attention for each head and the multihead highlighting . experimental results show that their proposed model significantly outperforms the baseline model .
Outcome:	The proposed model outperforms the baseline models on a multi-news dataset.

Copied to clipboard

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Programming Languages in NLP

Challenge:	Generating a readable summary that describes the functionality of a program is known as source code summarization.
Approach:	They propose a Transformer model that uses a self-attention mechanism to capture long-range dependencies by encoding source code tokens relative to the code token position.
Outcome:	The proposed model outperforms the state-of-the-art methods by a significant margin.

Dynamic Position Encoding for Transformers (2022.coling-1)

Copied to clipboard

Joyce Zheng, Mehdi Rezagholizadeh, Peyman Passban

Deep Attentive Sentence Ordering Network (D18-1)

Challenge:	In neural machine translation, the general task of translating is to reduce the input sentence into smaller units (also known as statistical phrases), select an optimal translation for each unit, and place them in the correct order.
Approach:	They propose a novel architecture that relies on a feed-forward backbone and self-attention mechanism to encode sequential/positional information.
Outcome:	The proposed architecture improves on multiple datasets in French, Italian, and German and shows that it is more efficient than the current model.

Copied to clipboard

Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang

Reasoning

Challenge:	Existing methods for sentence ordering tasks rely on linguistic knowledge and are domain specific.
Approach:	They propose a deep attentive sentence ordering network which integrates self-attention mechanism with LSTMs in the encoding of input sentences.
Outcome:	The proposed model outperforms the state-of-the-art models on Sentence Ordering and Order Discrimination tasks and is shown to be highly efficient.

Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)

Copied to clipboard

Martin Courtois, Malte Ostendorff, Leonhard Hennig, Georg Rehm

UniRel: Unified Representation and Interaction for Joint Relational Triple Extraction (2022.emnlp-main)

Challenge:	Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources.
Approach:	They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism.
Outcome:	The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%.

Copied to clipboard

Wei Tang, Benfeng Xu, Yuyue Zhao, Zhendong Mao, Yifeng Liu, Yong Liao, Haiyong Xie

Enhancing Rumor Detection Methods with Propagation Structure Infused Language Model (2025.coling-main)

Challenge:	Existing approaches to extract rich correlations between entities and relations are not fully exploited by existing methods.
Approach:	They propose to unify entities and relations by jointly encoding them within a concatenated natural language sequence and unify the modeling of interactions with a proposed Interaction Map.
Outcome:	The proposed method is more efficient and efficient than existing methods and can be scaled up to 2021.

Copied to clipboard

Chaoqun Cui, Siyuan Li, Kunkun Ma, Caiyan Jia

Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis (2021.acl-long)

Challenge:	Pretrained Language Models excel in various Natural Language Processing tasks, but performance on social media applications like rumor detection remains suboptimal.
Approach:	They propose a pretraining strategy to infuse information from propagation structures into pretrained language models to capture interactions of stance and sentiment crucial for rumor detection.
Outcome:	The proposed model outperforms existing methods on social media applications and significantly improves rumor detection performance.

Copied to clipboard

Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, Eduard Hovy

Sentiment Analysis

Challenge:	Existing methods to model relationships between aspects and opinion words are inefficient due to informal expressions and complexity of online reviews.
Approach:	They propose a dual graph convolutional networks model that considers complementarity of syntax structures and semantic correlations simultaneously.
Outcome:	The proposed model outperforms state-of-the-art methods on three public datasets and validates it.

LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020.emnlp-main)

Copied to clipboard

Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto

An Element-aware Multi-representation Model for Law Article Prediction (2020.emnlp-main)

Challenge:	Existing models for entity representations do not capture information in a knowledge base, and cannot represent entities that do not exist in the KB.
Approach:	They propose a pretrained contextualized representation of words and entities based on the bidirectional transformer.
Outcome:	The proposed model achieves impressive empirical performance on a wide range of entity-related tasks.

Copied to clipboard

Huilin Zhong, Junsheng Zhou, Weiguang Qu, Yunfei Long, Yanhui Gu

Representation Learning

Challenge:	Existing studies have shown that using law articles as external knowledge can improve the performance of the Legal Judgment Prediction.
Approach:	They propose a Law Article Element-aware Multi-representation Model which makes full use of law article information and can be used for multi-label samples.
Outcome:	The proposed model improves the accuracy of 5.84%, macro F1 of 6.42%, and micro F1 by 4.28% compared with baseline models like TopJudge.

Multi-resolution Annotations for Emoji Prediction (2020.emnlp-main)

Copied to clipboard

Weicheng Ma, Ruibo Liu, Lili Wang, Soroush Vosoughi

Visual Data in NLP

Challenge:	Emojis are able to express various linguistic components, such as emotions, sentiments, events, etc. emojis have the merit of preserving information more densely, compared to words, argues a new study.
Approach:	They propose to use passage-level and aspect-level emoji annotations to predict the proper emmojis associated with text.
Outcome:	The proposed method is heuristically generated and validated with a pre-trained BERT model.

What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020.emnlp-main)

Copied to clipboard

Yu-An Wang, Yun-Nung Chen

Scaling up the State Size of RNN LLMs for Long-Context Scenarios (2025.acl-long)

Challenge:	Existing work on pre-trained Transformers has focused on learning the meaning of positions . Embedding the position information in the self-attention mechanism is also an indispensable factor in NLP .
Approach:	They propose to use feature-level analysis to examine pre-trained Transformers' position embeddings . they also use empirical experiments to determine the appropriate positional encoding function .
Outcome:	The results of the empirical study can guide future work to choose the appropriate positional encoding function for specific tasks.

Copied to clipboard

Kai Liu, Jianfei Gao, Kai Chen

On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)

Challenge:	Existing RNN-based LLMs struggle with long-context scenarios due to their quadratic computational complexity and linear memory requirements.
Approach:	They propose an efficient scaling method to scale RNN models to match the 2k context length of Transformers with small parameters overhead.
Outcome:	The proposed method improves long-context understanding and improves performance on FDA recall-intensive tasks.

Copied to clipboard

Satwik Bhattamishra, Kabir Ahuja, Navin Goyal

A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (2023.findings-emnlp)

Challenge:	Existing studies on LSTMs have not revealed their ability to model syntactic properties.
Approach:	They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction.
Outcome:	The proposed model generalizes well on counter languages and its learned mechanism correlates with it.

Copied to clipboard

William Timkey, Tal Linzen

ARXSA: A General Negative Feedback Control Theory in Vision-Language Models (2025.findings-emnlp)

Challenge:	Theories of human sentence processing can be divided into two broad categories: expectation-based theories and memory-based ones.
Approach:	They propose to integrate expectations and retrieval from working memory into a unified cognitive model that can capture syntactic and semantic interference effects observed in human experiments.
Outcome:	The proposed model captures syntactic and semantic interference effects observed in human experiments.

Copied to clipboard

Zeyu Zhang, Tianqi Chen, Yuki Todo

Measuring the Mixing of Contextual Information in the Transformer (2022.emnlp-main)

Challenge:	a new approach to the self-attention mechanism is proposed for integrating data from multiple batches.
Approach:	They propose an autoregressive with exogenous inputs approach for the Transformer model . the proposed method transforms the Encoder block into a negative feedback predictive control system .
Outcome:	The proposed method is validated through comparative evaluations.

Copied to clipboard

Javier Ferrando, Gerard I. Gállego, Marta R. Costa-jussà

Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal (2022.emnlp-main)

Challenge:	Experimentally, we show that ALTI provides more faithful explanations and increased robustness than gradient-based methods.
Approach:	They propose to measure token-to-token interactions within each layer and then use them to aggregate model predictions.
Outcome:	The proposed method provides more faithful explanations and increased robustness than gradient-based methods.

Copied to clipboard

Byung-Doh Oh, William Schuler

Generalizing Backpropagation for Gradient-Based Interpretability (2023.acl-long)

Challenge:	Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism.
Approach:	They propose an entropy-based predictor that quantifies the diffuseness of self-attention and a distance-based one that captures the incremental change in attention patterns across timesteps.
Outcome:	The proposed models perform better over a rigorous baseline including GPT-2 surprisal than previous models that used entropy-based predictors and distance-based ones.

Copied to clipboard

Kevin Du, Lucas Torroba Hennigen, Niklas Stoehr, Alex Warstadt, Ryan Cotterell

Explainability & Interpretability in NLP

Challenge:	Several feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model’s output with respect to its inputs, but they reveal little about the inner workings of the model itself.
Approach:	They propose a generalized backpropagation algorithm that generalizes the gradient computation of a model to efficiently compute other interpretable statistics about the gradient graph of neural networks.
Outcome:	The proposed generalized algorithm can be used to compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy.

Instruction Position Matters in Sequence Generation with Large Language Models (2024.findings-acl)

Copied to clipboard

Yijin Liu, Xianfeng Zeng, Chenze Shao, Fandong Meng, Jie Zhou

Text Generation

Challenge:	Large language models (LLMs) can perform conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning.
Approach:	They propose to shift the position of task instructions after the input sentences to enhance the model's instruction-following capability.
Outcome:	The proposed method outperforms traditional settings across various model scales (1B / 7B & 13B) and different sequence generation tasks (translation and summarization) without any additional data or annotation costs.

Fine-Tuning Pre-trained Transformers into Decaying Fast Weights (2022.emnlp-main)

Copied to clipboard

Huanru Henry Mao

Time-aware Graph Neural Network for Entity Alignment between Temporal Knowledge Graphs (2021.emnlp-main)

Challenge:	Autoregressive Transformers incur O(T) complexity during per-token generation due to the self-attention mechanism.
Approach:	They propose a kernel-based method to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity.
Outcome:	The proposed method outperforms prior methods and retains 99% of attention’s performance on WikiText-103 against more complex attention substitutes.

Copied to clipboard

Chengjin Xu, Fenglong Su, Jens Lehmann

TAGPRIME: A Unified Framework for Relational Structure Extraction (2023.acl-long)

Challenge:	Existing embedding-based approaches disregard time information that exists in many large-scale knowledge graphs, leaving much room for improvement.
Approach:	They propose a Time-aware Entity Alignment approach that incorporates relation and time information into a vector space and uses Graph Neural Networks to learn entity representations.
Outcome:	The proposed approach outperforms the state-of-the-art methods on real-world TKG datasets due to the inclusion of time information.

Copied to clipboard

I-Hung Hsu, Kuan-Hao Huang, Shuning Zhang, Wenxin Cheng, Prem Natarajan, Kai-Wei Chang, Nanyun Peng

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)

Challenge:	Existing models for natural language processing (NLP) do not address common tasks.
Approach:	They propose to take a unified view of all the tasks and introduce a model that appends priming words about the condition to the input text.
Outcome:	The proposed model is based on ten datasets across five different languages and covers ten tasks that cover ten languages.

Copied to clipboard

Jungmin Yun, Mihyeon Kim, Youngbin Kim

Information Retrieval

Challenge:	Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance.
Approach:	They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands.
Outcome:	Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model.

Computation Mechanism Behind LLM Position Generalization (2025.acl-long)

Copied to clipboard

Chi Han, Heng Ji

Challenge:	Existing studies have explored how LLMs handle positional relevance, but how they handle it remains unexplored.
Approach:	They propose to enforce certain computational mechanisms to allow for the tolerance in position perturbations in large language models (LLMs) they also find a pattern in intermediate features that allows this effect to be observed .
Outcome:	The proposed models can understand text with position perturbations and generalize to longer sequences than those seen during training with the latest techniques.

LLoCO: Learning Long Contexts Offline (2024.emnlp-main)

Copied to clipboard

Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa

Natural Language Interfaces

Challenge:	Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation.
Approach:	They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA.
Outcome:	The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering.

LongAttn: Selecting Long-context Training Data via Token-level Attention (2025.findings-acl)

Copied to clipboard

Longyun Wu, Dawei Zhu, Guangxiang Zhao, Zhuocheng Yu, Junfeng Ran, Xiangyu Wong, Lin Sun, Sujian Li

Responsible & Trustworthy NLP

Challenge:	Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency.
Approach:	They propose a token-level framework which quantifies long-range dependencies for LLMs by calculating token-based dependency strength and distribution uniformity of token scores.
Outcome:	The proposed framework quantifies long-range dependencies, enabling more accurate and efficient data selection.

MSCFFN: A New FFN with Multi-Space Cross to Accelerate Transformer (2023.findings-emnlp)

Copied to clipboard

Tang Dongge, Qing Yang

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling (2026.acl-long)

Challenge:	Existing models to reduce computation complexity are limited in some areas . a new structure to reduce the computation complexity is proposed to accelerate Transformers .
Approach:	They propose a new feed forward network structure which splits matrix space to smaller space to reduce computation complexity.
Outcome:	The proposed model can achieve a faster speed and better accuracy on the long-range arena benchmark.

Copied to clipboard

Yujie Chen, Tailai Chen, Yifeng Gao, Zoe Wanying He, Yijue Xu, Shaobo Wang, Linfeng Zhang

Green & Sustainable NLP

Challenge:	Existing methods to reduce sequence length rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention.
Approach:	They propose a method that selectively halts stabilized tokens by monitoring layer-wise update dynamics of the self-attention mechanism.
Outcome:	The proposed method can reduce prefill complexity while preserving model accuracy and hardware efficiency.

Length-Induced Embedding Collapse in PLM-based Models (2025.acl-long)

Copied to clipboard

Yuqi Zhou, Sunhao Dai, Zhanshuo Cao, Xiao Zhang, Jun Xu